ENTROPIC PROJECTIONS AND DOMINATING POINTS 



CHRISTIAN LEONARD 



Abstract. Entropic projections and dominating points are solutions to convex mini- 
mization problems related to conditional laws of large numbers. They appear in many 
areas of applied mathematics such as statistical physics, information theory, mathemat- 
ical statistics, ill-posed inverse problems or large deviation theory. By means of convex 
conjugate duality and functional analysis, criteria are derived for the existence of entropic 
projections, generalized entropic projections and dominating points. Representations of 
the generalized entropic projections are presented. It is shown that they are the "mea- 
sure component" of some extended entropy minimization problem. This approach leads 
to new results and offers a new point of view. It also permits to extend previous results 
on the subject by removing unnecessary topological restrictions. As a by-product, new 
proofs of already known results are provided. 
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1. Introduction 

Entropic projections and dominating points are solutions to convex minimization prob- 
lems related to conditional laws of large numbers. They appear in many areas of applied 
mathematics such as statistical physics, information theory, mathematical statistics, ill- 
posed inverse problems or large deviation theory. 



Conditional laws of large numbers. Suppose that the empirical measures 

1 n 

L n :=-Y^8 Zi , 7i >1, (1.1) 

i=l 

of the Z- valued random variables Z-y, Z 2 , ■ ■ ■ {S z is the Dirac measure at z) obey a Large 
Deviation Principle (LDP) in the set Pz of all probability measures on Z with the rate 
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function I. This approximately means that P(L n G A) x exp[— nmfp e ^I(P)} for 

n— >oo 

^4 C Pz- With regular enough subsets A and C of Pz, one can expect that for "all" A 

( 1 if .4 =3 P 
lim P(L n G -4 | L n G C) = I ' \C * 
n^oo v 1 ' \ 0, otherwise 

where P* is a minimizer of I on C To see this, remark that (formally) F(L n G A \ L n G 
C) x exp[— n(infp e ^nc PyP) — inf p e c I(P))). If / is strictly convex and C is convex, P* 

n— >oo 

is unique and this roughly means that conditionally on L n G C, as n tends to infinity L n 
tends to the solution P* of the minimization problem 

minimize I(P) subject to P G C, P G P z (1-2) 

Such conditional Laws of Large Numbers (LLN) appear in information theory and in 
statistical physics where they are often called Gibbs conditioning principles (see [H Sec- 
tion 7.3] and the references therein). If the variables Zi are independent and identically 
distributed with law R, the LDP for the empirical measures is given by Sanov's theorem 
and the rate function I is the relative entropy 

I(P) = I(P\R) = J log dP, P G Pz. 

Instead of the empirical probability measure of a random sample, one can consider an- 
other kind of random measure. Let zi,z 2 ,... be deterministic points in Z such that 
the empirical measure - Y^i=i ^ converges to R G Pz- Let W±, W 2 , ... be a sequence of 
independent random real variables. The random measure of interest is 

1 n 

L n = -Y^Wi5 z . (1.3) 

8=1 

where the W^s are interpreted as random weights. If the weights are independent copies 
of W, as n tends to infinity, L n tends to the deterministic measure KW.R and obeys the 
LDP in the space Mz of measures on Z with rate function I(Q) = j z 7*(^§) dR, Q G M z 
where 7* is the Cramer transform of the law of W. In case the W^s are not identically 
distributed, but have a law which depends (continuously) on z i: one can again show that 
under additional assumptions L n obeys the LDP in Mz with rate function 

I{Q) = {Lit{%{z))R{dz), UQ^R QeMz (L4) 

[ +00, otherwise 

where 7* is the Cramer transform of W z . As 7* is the convex conjugate of the log-Laplace 
transform of W, it is a convex function: / is a convex integral functional. It is often called 
an entropy. Again, conditional LLNs hold for L n and lead to the entropy minimization 
problem: 

minimize I{Q) subject to Q G C, Q G Mz (1.5) 

The large deviations of these random measures and their conditional LLNs enter the 
framework of Maximum Entropy in the Mean (MEM) which has been studied among 
others by Dacunha-Castelle, Csiszar, Gamboa, Gassiat, Najim see [5j El [101 [21] and also 
[3, Theorem 7.2.3]. This problem also arises in the context of statistical physics. It has 
been studied among others by Boucher, Ellis, Gough, Puli and Turkington, see [H[9]. 

The relative entropy corresponds to 7*(t) = t\ogt — t + 1 in (j 1.4ft : the problem (11.2ft is 
a special case of fl 1.5ft with the additional constraint that Q(Z) = 1. 



ENTROPIC PROJECTIONS AND DOMINATING POINTS 



3 



In this paper, the constraint set C is assumed to be convex as is almost always done in 
the literature on the subject. This allows to rely on convex analysis, saddle-point theory 
or on the geometric theory of projection on convex sets. 

Entropic projections. The minimizers of (11.51) are called entropic projections. It may 
happen that even if the minimizer is not attained, any minimizing sequence converges to 
some measure which does not belong to C. This intriguing phenomenon was discovered 
by Csiszar [3J. Such a is called a generalized entropic projection. 

In the special case where / is the relative entropy, Csiszar has obtained existence results 
in [2] together with dual equalities. His proofs are based on geometric properties of the 
relative entropy; no convex analysis is needed. Based on the same geometric ideas, he 
obtained later in [3] a powerful Gibbs conditioning principle for noninteracting particles. 
For general entropies as in (jl.4p . he studies the problem of existence of entropic and 
generalized entropic projections in [3]. 

The minimization problem (jl.5p is interesting in its own right, even when conditional 
LLNs are not at stake. The literature on this subject is huge. Some bibliographical entries 
are given in [15J. 

Dominating points. Let the constraint set C be described by 

C = {Q G M Z ;TQ E C} (1.6) 

where C is a subset of a vector space X and T : Mz — > X is a linear operator. As 
a typical example, one can think of TQ = j z 9(z) Q(dz) where 9 : Z — > X is some 
function and the integral should be taken formally for the moment. With L n given 
at (11.11) or (II. 3p . if T is regular enough, we obtain by the contraction principle that 
X n := TL n = lY.txQ(Zi) e X or X n := TL n = ~ £T =1 W^Zi) obeys the LDP in 
X with rate function J(x) = inf{/(Q);Q G Mz,TQ = x}, x G X. Once again, the 
conditional LLN for X n is of the form: For "all" A C X, 

limP(X„eA|X„ eC ) = {j; *J££ 

where solution to the minimization problem 

minimize J(x) subject to x G C, x G X (1.7) 

The minimizers of (ll.7p are called dominating points. This notion was introduced by Ney 
(221 [23] in the special case where (Zj)j>i is an iid sequence in Z = R d and 6 is the identity, 
i.e. X n = - Y17=i ^i- Later, Einmahl and Kuelbs [HI [32] have extended this study to a 
Banach space Z. In this iid case, J is the Cramer transform of the law of Z\. 

Presentation of the results. One treats the problems of existence of entropic projec- 
tions and dominating points in a unified way, taking advantage of the mapping TQ = x. 
Although it is simple, this connection does not seem to be used in previous literature. 
Hence, one mainly concentrates efforts on the entropic projections and then transports 
the results to the dominating points. 

It will be proved at Proposition 15.51 that the entropic projection exists on C if the 
supporting hyperplanes of C are directed by sufficiently integrable functions. In some cases 
of not enough integrable supporting hyperplanes, the representation of the generalized 
projection is still available and given at Theorem 15.61 It will appear that the generalized 
projection is the "measure" part of the minimizer of an extended minimization problem. 

For instance, with the relative entropy I(.\R), the projection exists in C if its supporting 
hyperplanes are directed by functions u such that f z e a '"' dR < oo for all a > 0, see 
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Proposition 15.101 If these u only satisfy j z e a '"' dR < oo for some a > 0, the projection 
may not exist in C, but the generalized projection is computable: its Radon- Nykodym 
derivative with respect to R is characterized at Proposition 15.131 

One finds again some already known results of Csiszar [H 0], U. Einmahl and Kuelbs 
[8lfl3] with different proofs and a new point of view. The representations of the generalized 
projections are new results. The conditions on C to obtain dominating points are improved 
and an interesting phenomenon noticed in [13J is clarified at Remark 16.91 by connecting it 
with the generalized entropic projection. 

The main results are Theorems 14.11 14.71 15.61 and 16.81 



Outline of the paper. At Section [21 one gives a precise f ormu lation of the entropy 



minimization problem (11.51) and a natural extension of it, see (]Pcv at Section I2T31 Then 



one recalls at Theorems 12.141 and 12 . 1 71 results from [15J about the existence and uniqueness 



of the solutions of (jl.5p and (Pg), related dual equalities and the characterizations of their 
solutions in terms of integral representations. 

Examples of standard entropies and constraints are presented at Section [3j 

One shows at Theorem 14.71 in Section H] that under "bad" constraints (this notion is 
specified at Section 12.41 in terms of the set C and the operator T appearing in (II. 6p ). 
although the problem ( 11. 5p may not be attained, its minimizing sequences may converge 
in some sense to some measure Q* : the generalized entropic projection. 

Section [5] is mainly a restatement of Sections [2] and H] in terms of entropic projections. 
The results are also stated explicitly for the special important case of the relative entropy. 

Section [6] is devoted to dominating points. As they are continuous images of entropic 
projections, the main results of this section are corollaries of the results of Section [51 

Notation. Let X and Y be topological vector spaces. The algebraic dual space of X is 
X*, the topological dual space of X is X' . The topology of X weakened by Y is cr(X, Y) 
and one writes (X, Y) to specify that X and Y are in separating duality. 
Let / : X — > [— oo, +oo] be an extended numerical function. Its convex conjugate with 
respect to (X,Y) is f*(y) = sxvp xeX {(x, y) — f(x)} G [— oo,+oo], y G K Its sub differential 
at x with respect to (X, Y) is d Y f{x) = {y G Y; f(x + £) > f(x) + (y,OM G X}. If no 
confusion occurs, one writes df(x). 

Let A be a subset of X, its intrinsic core is icor A = {x G A; W G afL4, 3t > 0, [x,x + 
t(x' — x)[c A} where afL4 is the affine space spanned by A. Let us denote dom/ = {x G 
X; f(x) < oo} the effective domain of / and icordom/ the intrisic core of dom/. 
The indicator of a subset A of X is defined by 

L ( x \ _ / °> if x e A x e x 
tA{X) < otherwise ' X 

One writes 

I(p(u) := / (f(z,u(z)) R(dz) = / <p(u) dR 
J z J z 

and / = J 7 * for short, instead of (11.41) . 



2. Minimizing entropy under convex constraints 
In this section, the main results of [TH] are recalled. 
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2.1. Orlicz spaces. The fact that the generalized projection may not belong to C is 
connected with some properties of Orlicz spaces associated to /. Let us recall some basic 
definitions and results. 

A set Z is furnished with a a-fmite nonnegative measure i?ona ex-field which is assumed 
to be i?-complete. A function p : Z x R is said to be a Young function if for i?-almost 
every z, p(z, •) is a convex even [0, oo]-valued function on R such that p(z, 0) = and 
there exists a measurable function z i— > s z > such that < p(z, s z ) < oo. 
In the sequel, every numerical function on Z is supposed to be measurable. 

Definitions 2.1 (The Orlicz spaces L p and E p ). The Orlicz space associated with p is 
defined by L p = {u : Z — > R; \\u\\ p < +00} where the Luxemburg norm || • || p is defined by 
\\u\\ p = inf {f3 > ; j z p(z , u(z) / ' 0) R(dz) < l} and R-a.e. equal functions are identified. 
Hence, 

L p = |w : Z — > R ; 3a a > 0, J p(^z, a u(z)j R(dz) < 00 1 . 
A subspace of interest is 

E p = ju : Z -> R ;Va > 0, ^ p^,a«(z)j i?(tte) < ooj . 

Of course E p C L p . Note that if p doesn't depend on 2 and p(s ) — 00 f° r some s Q > 0, 
E p reduces to the null space and if in addition R is bounded, L p is L^. On the other 
hand, if p is a finite function which doesn't depend on z and R is bounded, E p contains 
all the bounded functions. 

Duality in Orlicz spaces is intimately linked with the convex conjugacy. The convex 
conjugate p* of p is defined by p*(z, t) = sup seR {st — p(z, s)}. It is also a Young function 
so that one may consider the Orlicz space L p *. 

A continuous linear form £ e L' p is said to be singular if for all u G L p , there exists a 
decreasing sequence of measurable sets (A n ) such that R(n n A n ) = and for all n > 1, 
(£,ulz\A n ) — 0. Let us denote L s p the subspace of L' p of all singular forms. 

Theorem 2.2 (Representation of E' and L' ). 

(a) Suppose that p is a finite Young function. Then, 

(i) E' is isomorphic to L p *; 

(ii) £ G L' p is singular if and only if (£, u) = 0, for all u in E p . 

(b) Let p be any Young function. Any £ G L' p is uniquely decomposed as 

£ = t + t (2.3) 
with £ a G L p *R and t G L s . This means that L' p is the direct sum L' p = L p *R © L s . 

We denote the space of i?-absolutely continuous signed measures having a density in 
the Orlicz space L p * by L P *R. 

Proof. For a proof of (a), see [TTJ Thm 4.8] and [121 Proposition 2.1]. 
About (b): When L p = L^ this result is the usual representation of L'^. When p is a 
finite function, this result is ([12], Theorem 2.2). The general result is proved in [23], with 
p not depending on z but the extension to a ^-dependent p is obvious. □ 

In the decomposition (I2.3p . £ a is called the absolutely continuous part of £ while £ s is 
its singular part. £ a is a measure which is absolutely continuous with respect to R and £ s 
is not a measure: it is additive but not cr-additive. 
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Definition 2.4 (A 2 -condition). The function p is said to satisfy the A 2 -condition if there 
exist C > 0, s Q > such that for all s > s Q , p(2s) < Cp(s). 

When R is bounded, in order that E p = L p , it is enough that p satisfies the A 2 -condition. 
If R is unbounded, for E p = L p , it is enough that the A 2 -condition holds globally, i.e. 
with s Q = 0. Consequently, if p satisfies the A 2 -condition we have L' p = L p *R so that L s p 
reduces to the null vector space. 

2.2. The entropy minimization problem (Pc)» 

Entropy. Let R be a positive measure on a space Z and take a [0, oo]-valued measurable 
function 7* on Z x R such that 7* (z, •) := 7* is convex and lower semicontinuous for all 
z G Z. Denote Mz the space of all signed measures Q on Z. The entropy functional to 
be considered is defined by 



[ +00 otherwise 



(2.5) 



where Q -< R means that Q is absolutely continuous with respect to R. Assume that for 
each z there exists a unique m(z) which minimizes 7* with 

j* z (m(z)) = 0,\/ze Z. 

Then, / is [0, oo]-valued, its unique minimizer is mR and I(mR) = 0. 

Relevant Orlicz spaces. Since 7* is closed convex for each z, it is the convex conjugate of 
some closed convex function 7 2 . Defining 

X(z, s) = j(z, s) — m(z)s, z G Z, s G R, 

one sees that for R-a.e. z, X z is a nonnegative convex function and it vanishes at 0. Hence, 

\o(z, s) = max[A(,2, s), X(z, —s)] G [0, 00], z G Z, s G R 

is a Young function and one can consider the corresponding Orlicz spaces L\ and L\* 
where A*(z, •) is the convex conjugate of X (z, •). 

The effective domain of / is included in mR + L\*R. It will be assumed from now on that 
m G L\* so that dom I = L\*R. 

Constraint. In order to define the constraint, take X a vector space and a function 
9 : Z — > X Q . One wants to give some meaning to the formal constraint j z 8 d£ = x with 
£ G L\*R and x G X Q . Suppose that X Q is the algebraic dual space of some vector space 
y o and define for all y G y o and z G Z, (y, 0{z))y Ot x o , Assuming that 

(y ,e(-))cL Xo , (2.6) 

Holder's inequality in Orlicz spaces allows to define the constraint operator I G L' x<> 1— > 



{9,1) G X by: (y, (6,1)) = Uy,0)J) ,Vy G y o . If i G L K R, one sometimes 

write (8, £) — f z 9 di. One sometimes denote 

Te = (6,e), eeL' Xo . 

Minimization problem. We are now ready to state the minimization problem (jl.5p pre- 
cisely: 

minimize I(Q) subject to J 9dQ EC, Q G L\*R (P c ) 
where C is a convex subset of X a . 
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2.3. The extended entropy minimization problem(Pc). The extended entropy is 
defined by 

I(£) = i(r) + F(r), eeL' (2.7) 



where, using the notation of Theorem 12.21 

I%n = 4^(0 = sup G L Xo ,I,(u) < 00} g [0,oo]. 

The associated extended minimization problem is 

minimize I(£) subject to (9, £) G C, £ G L' K (P c ) 

/ is the greatest convex c(L' x<> , L Ao )-lower semicontinuous extension of I to L' x<> D L X *R, 
see [15] for references about this result. 



2.4. Good and bad constraints. If the Young function A<> doesn't satisfy the A2- 
condition (see Definition 12. 4p . for instance if it has an exponential growth at infinity as 
in (13. ip below, the small Orlicz space E x<> may be a proper subset of L\ a . Consequently, 
for some functions 9, the integrability property 

(y o ,9(-))GE Xa (2.8) 

or equivalently 

Wy G y o , £ X(z, (y, 9(z))) R(dz) < 00 (A$) 
may not be satisfied while the weaker property (12. 6p : (y o ,9(-)) C L Xt> , or equivalently 

Wy G y o , 3a > 0, £ \{z, a(y, 9{z))) R{dz) < 00 (Af) 
holds. In this situation, an alyti cal complications occur (see Section H]). This is the reason 



why constrai nt sa tisfying (Ag) are called good constraints, while constraints satisfying 



(Ag) but not (Ag) are called bad constraints. 



One denotes y^ the subset of X* which is isomorp hic t o the strong closure of the 



subspace {(y,9);y G y o } in L Xo . Under the assumption (Ag), for all y G y o , (y,9) G E x 
and y^ = yE '■ the strong closure of {(y,9);y G y o } in E\ a 



Under the assumption (A^), the convex set C is said to be a good constraint set if 
T- 1 CHL xl R= P| \fReL K R- [ (y,9)fdR>a y ) for some Y C y E (2.9) 



and some function ?/ G K 1— > a y G JR.. Under the assumption (Ag), it is said to be a 6ad 
constraint set if 

T^C = f| {£ G L' Ao ; > a y } for some F C y L (2.10) 

The se special shapes (12.91) and (12.101) imply some closure property of C. For comparison, if 
(Ag ) holds and C is only supposed to be convex, T~ l C = f]r y a - )gA {£ G L' Ao ; ((y, 9),£) > a} 



with ic^xi 
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2.5. Seve ral function spaces and cones. To state the extended dual problems ( Dp ) 
and ( Dc ) below, notation is needed. If A is not an even function, one has to consider 

\+(z,s) = X(z,\s\) /- 911 \ 
X4z,s) = X(z,-\s\) [Z - ii} 

which are Young functions and the corresponding Orlicz spaces. 

Let E be a Riesz vector space for the order relation < . Any e G E admits a positive 
part: e + := e V and a negative part: e_ := (— e) V 0. One writes |e| = e + + e_. There is 
a natural order on the algebraic dual space E* of a Riesz vector space E which is defined 
by: e* < f* 4^ (e*, e) < (/*, e) for any e E E with e > 0. A linear form e* G £7* is said to 
be relatively bounded if for any / G E, f > 0, we have sup e .| e |<y |(e*, e}| < +oo. Although 
E* may not be a Riesz space in general, the vector space E b of all the relatively bounded 
linear forms on E is always a Riesz space. In particular, the elements of E b admit a 
decomposition in positive and negative parts e* = e* + — e*_. 

Definitions 2.12. For any relatively bounded linear form ( on L' x<> i.e. ( 6 L Xo , one 
writes: 

• C e K'l to specify that £±, L , n£ # G L" x 

A ± A« =c 

• ( G to specif tfmi C±|l.. ma e ^Ai 

I A_j_ A<> ± 

• QeK x to specify that £±i L G L A± 



• C e ^a* to specify that C±i L G L A * 

• C G f^' to specify that (±> L s nL , G Lf 

where A± are defined at 112.11}) and (±\ L±nL > x G Lj_ means that the restriction of (± to 

L± fl L' x<> is continuous with respect to relative topology generated by the strong topology 
of L± on L± n L' Xo . 

(1) The sets K x , K x „, K\, K x * and K x are defined to be the corresponding subsets of 
L' x . They are not vector spaces in general but convex cones with vertex 0. 

(2) The a (K , K' x )- closure A of a set A is defined as follows: ( G L Xt> is in A if 
C±\L' nM %s m the a ( L x+ n L A > L 'x+ n L' x )-closure of A ± = {(±; ( G A}. Clearly, 

l A_j_ Ao - 1 - - 1 - 

4 = {C±;(eI}. 

One defines similarly the a(K' x „, K x *), a(K x ,K' x ), a(K s x ,,Kx*) and a(K x , En- 
closures. 

(3) Let A be a subset of L\ a . Its strong closure s-cl A in K\ is the set of all measurable 
functions u such that u± is in the || • \\\ ± -closure of A± = {v±; v G A}. 

Let p be a Young function. By Theorem 12.2} we have L" p = [L p .R © L s p ] © L s p '. For any 
£ G L p = (L P *R © L s p )', let us denote the restrictions = (\l p ,r and (2 = Since, 
(Lp*R)' ~ L p © L p *, one sees that any ( G L" p is uniquely decomposed into 

C = + Cf + C2 (2.13) 

with Ci = C? + Cf e L' p *, d a G L p , (I G L s p * and C2 G L*'. With our definitions, K'{ = 
[K x ®K S X J® K s x ' and the decomposition fl2T3|) holds for any C G if" with 

Ci = Ci a + C!e/^©^, 

C2 G Jtf- 
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2.6. The assumptions. Let us first collect the assumptions on R, 7* and 9. 
Assumptions (A). 

(Ar) It is assumed that the reference measure R is a cx-finite nonnegative measure on a 

space Z endowed with some .R-complete a-field. 
(A 7 ») Assumptions on 7*. 

(1) 7*(-,t) is z-measurable for all t and for .R-almost every z G Z, j*(z,-) is 
a lower semicontinuous strictly convex [0, +oo]-valued function on R which 
attains a unique minimum. Let m(z) denote this minimizer. 

(2) It is also assumed that for .R-almost every z G Z, the minimum value is 
j*(z,m(z)) = and there exist a(z),b(z) > such that < r y*(z,m(z) + 
a(z)) < 00 and < j*(z, m(z) — b(z)) < 00. 

(3) J z \*(z, am(z)) R(dz) + f z X*(z, —am(z)) R(dz) < 00, for some a > 0. 

(4) For .R-almost every z G Z, lim^-boo 7* (t) jt = +00. 
(Ae) Assumptions on 9. 

(1) for any y G y o , the function z G Z \— > (y, 9(z)) G M. is measurable; 

(2) for any y G y o , (y, 9(-)) = 0, .R-a.e. implies that y = 0; 

(3) My G y o , 3a > 0, j z X(z, a(y, 9{z))) R{dz) < 00. 



2.7. The solution of (Pc)- The dual problem associated with (P^) is 

maximize inf (y,x) - lJ{y,9)), yey L (D c ) 

and the extended dual problem associated with QPcD is 

maximize inf (uj,x) — I y ((uj,9)), to G y (^c) 

where y is the convex cone of all linear forms uj on X such that (u, 9{-))x* ,x is in the 
o~(K\, iT A .)-closure of {(y,8);y G y o }. 
Let us define 

T*(x) = sup < (y,x) - / "/((y,9)) dR\ , x G X Q 
yey I J 

the convex conjugate of 

r(i/) = Jj{(y,6))dR, y ey . 

Theorem 2.14 (Problem (Pc))- Suppose that 

(1) the assumptions (A) and (A e ) are satisfied; 

(2) C satisfies JO) . 

Then: 

(oj The following dual equality for QPgD holds: 

inf(P c ) = sup(D c ) = sup(D c ) = inf T*(x) G [0, 00]. 



(b) If C DdomT* 7^ or equivalently CflTdom/ 7^ 0, £/ien QPgD admits a unique solution 
Q in L\*R and any minimizing sequence (Q n )n>i converges to Q with respect to the 
topology a(L x *.R, E\ ). 

Suppose that in addition C fl icordomT* 7^ or equivalently C D icor (Tdom I) 7^ 0. 
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(c) Let us define x = J 2 6 dQ in the weak sense with respect to the duality (y o , X a ). There 
exists uj G y such that 

(a) x G C n domT* 

(6) (&,x) XStXo < (l>,x) x *, Xo ,Vx eCn domT* (2.15) 
(c) Q(dz)= 1 >((u,9(z)))R(dz). 



Furthe rmor e, Q G L X *R and uj ay satisfy $2.15\) if and only if Q solves QPc| ) and uj 
solves (Dp). 
(d) Of course, \2.lh\ c) implies 

x = J 6i({u,e))dR (2.16) 

in the weak sense. Moreover, 

1. x minimizes V* on C, 

2. I{Q) = T*(x) = f z j*o 6))dR<oo and 

3. I{Q) + J zl ((u,9))dR = J z (Cu,9)dQ. 

Proof. This result is [15| Theorem 5.2]. □ 

Following the terminology of Ney [22l [23] . as it shares the properties (a), (b) and (12.161) . 
the minimizer x is called a dominating point of C with respect to T* (see Definition 16.21 
below) . 

2.8. The solutions of (Pc)- Let us turn our attention to the extended problem (Pc)- 
Denoting the operator T : £ G L' x<> i— > (6, i) G X , it is shown in [Po] that TL' X<> C Xl 
where X L C X Q is the topological dual space of (3^,1 • \l) with \y\ L = \\(y, 9(-))\\l x ■ 
Hence T : L' x — > and one can define its adjoint T* : <Y£ — > L x<> for all u; G X£ by: 
(£, T*co) L>x ^ = (Tl, co) Xl>X£ ,W G L^. We have the inclusions y o C y L C X*. 
The extended dual problem is 

maximize inf (u>, x) — I~(\T*u]^) — ld(\T*u>]o), uj G y (Dc) 

where .D is the cr(K x , if|)-closure of dom/ 7 and 3^ is the cone of all uj G X£ such that 
T*cj G K'l and [T*u;]f = 0. 

As R is assumed to be a- finite, there exists a measurable partition (Z k )k>i of Z : \_\ k Z k = 
Z, such that R(Z k ) < oo for each k > 1. 

Theorem 2.17 (Problem (Pc))- Suppose that 

(1) the assumptions (A) are satisfied; 

(2) C satisfies tZlW : 

(3) for each k > 1, L x<> (Z k , R\ Zk ) is dense in L x+ (Z k , R\ Zk ) and L x _(Z k ,R\ Zk ) with 
respect to the topologies associated with \\ ■ \\\ + and \\ ■ \\ x _. 

Then: 

(a) The following dual equality for (Pc) holds: 

inf (P c ) = inf T*(x) = sup(D c ) = sup(D c ) G [0, oo]. 



(b) If C DdomT* ^ or equivalently CflTdom/ ^ 0, then (Pc) admits solutions in L' x<> , any 
minimizing sequence admits o~(L' x , L x<! )- cluster points and every such point is a solution 

to dQ. 



ENTROPIC PROJECTIONS AND DOMINATING POINTS 11 

Suppose that in addition we have 

Cf] icordomT* ^ 
or equivalently C H icor (Tdomf) 7^ 0. Then: 



(c) Let £ be a solution to (P_c) and denote x — T£. There exists u G y such that 

(a) xGCndomP 

(6) (oj,x) x£>Xl < (u),x) x * iXl ,Vx eCndomP (2.18) 
(c) Ui z {[T*Q]f)R + D^{[T*u] 2 ) 

where 

D ± ( V ) = {£ G L 8 Xo ;Vh G L Ao , + h G D => (h,£) < 0} 
is the outer normal cone of D at rj. 

T*uj is in the o~(K", K' x ) -closure ofT*(domT) and there exists some u G X* such that 

pro]* = <«.»(•)> w 

is a measurable function in the strong closure of T* (domT) in K\. 

Furthermore, £ G L' x and uo G y satisfy A2.18\) if and only if £ solves (P_c) and u> solves 

E3- 

(d) Of course, h2.18l c) implies 

x = J 9i{{u,6))dR+ (9,t). (2.19) 

Moreover, 

1. x minimizes V* on C, 

2. 7(1) = T*(x) = J zl *°l'((Z,e))dR+ (T*u,£ s ) K s, !Ki < oc, 

3. I(t) + J z -f((u,9))dR = f z (u,9)di a and 

4. I(t) = sup{(w, I s ) ;u G dom/ 7 } = (T*u;, 

Proof. This result is [T5| Theorem 4.6] which is proved without assuming (At*). □ 

Proposition 2.20. For the assumption (3) of Theorem 2.11 to be satisfied, it is enough 
that one of these conditions holds 

(i) A is even or more generally < liminf^oo -r-(t) < limsup^^ y-(t) < +oo; 

(ii) lim^oo y-(t) = +oo and A_ satisfies the A2- condition \2.1$ . 

Proof. This result is [T5| Proposition 4.9]. □ 



Remark 2.21. Since it is assumed that x G icordomT*, no infinite force field (see 
for this notion) enters the dual representation of l x . If is the left bound of dom7*, 
one has Y(— 00) = and in case x G domT* \ icordomT*, it may happen that (y x ,6) 
takes (formally) the value —00 on some subset Z of Z so that vanishes on Z Q . A 
similar phenomenon occurs if dom 7* admits a finite right bound k z on a set Z K such that 
R(Z K ) > 0. Then, 7^+00) = k z < 00 and (z) = k z on Z K . Tor the details, see [IB] . 

3. Some examples 

3.1. Some examples of entropies. Important examples of entropies occur in statistical 
physics, probability theory and mathematical statistics. Among them the relative entropy 
plays a prominent role. 
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Relative entropy. The reference measure R is assumed to be a probability measure. The 
relative entropy of Q £ M z with respect to R £ Pz is 

I(QI^) = i ^ log (S dQ if ? ^ * and Q e Pz , Qe M Z . 

y +00 otherwise 

{tlogt-t+1 if t > 
1 if t = , m(z) = 1 and 

+00 if t < 

A,(s) = e s - s - 1, s£M,^£Z. (3.1) 

^4 variant. Taking the same 7* and removing the unit mass constraint gives 

H(Q\R) = l /*[§ 1 °g(&)-§ + 1 ] dR H0<Q^R 
\ +00, otherwise 

This entropy is the rate function of (11.31) when (Wj)j>i is an iid sequence of Poisson(l) 
random weights. If R is cr-finite, it is the rate function of the LDP of normalized Poisson 
random measures, see 



Extended relative entropy. Since A(s) = e s — s — 1 and R £ Pz is a bounded measure, we 
have A (s) = r(s) := e' s ' — |s| — 1 and the relevant Orlicz spaces are 

l/l log l/l dH< 00} 





If 


E T = 


{u 


L T = 


{u 



The extended relative entropy is defined by 

I(e\R) = I '{t\R)+ sup U£ s ,u);u, J e u dR< ooj , £ £ £{Z) (3.2) 

where i = l a + I s is the decomposition into absolutely continuous and singular parts of I 
in L; = L T * © LJ, and 6(2) = {t £ L' T ; t > 0, (£, 1) = 1}. Note that 6(2) depends on R 
and that for all I £ 6(2), l a £ Pg fl L T *R. 

3.2. Some examples of constraints. Let us consider the two standard constraints 
which are the moment constraints and the marginal constraints. 

3.2.1. Moment constraints. Let 9 = (9k)i<k<K be a measurable function from 2 to X = 
K. . The moment constraint is specified by the operator 

thu [ I e k di) £ r k , 

Z J Kk<K 



which is defined for each i £ Mz which integrates all the real valued measurable functions 
6 k . The adjoint operator is 



yfc^fc, y — vyi) • • • 5 c ^ A • 

Kfc<K 
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Marginal constraints. Let Z = Ax B be a product space, Mab be the space of all bounded 
signed measures on AxB and Uab be the space of all measurable bounded functions u 
on AxB. Denote £a = £{■ x B) and £b = £{A x •) the marginal measures of £ G Mab- 
The constraint of prescribed marginal measures is specified by 

/ 6d£ = (£ a ,£b) e M a x M b , £e M AB 

J AxB 

where Ma and Mb are the spaces of all bounded signed measures on A and B. The 
function 6 which gives the marginal constraint is 

9(a, b) = (S a , S b ), a e A,b e B 

where 5 a is the Dirac measure at a. Indeed, (£a,£b) = Jaxb^ -' ^(dadb). 
More precisely, let Ua, Ub be the spaces of measurable functions on A and B and take 
3^o = Ua x Ub and X Q = XJ A x U%. Then, 9 is a measurable function from Z = AxB to 
X — U*a x U% and the adjoint of the marginal operator 

(6,£) = (£ A ,£B)eU* A xU* B , £eU* AB 

where (/, I a) := (/ ® 1, £) and (g, £ B ) ■= (1 ® (?, ^) for all f e Ua and g E U B , is given by 

((f,9),6) = f®geU AB , feU A ,geU B (3.3) 

where / © g-(a, 6) := /(a) + g(b), a e A,b e B. 



4. Solving (P c ) with bad constraints 

In this section , th e minimization problem (Pc) is considered when the constraint func- 
tion ^ satisfies (Aj) but not necessarily (Ag). This means that the constraint is bad. 



Problem QPcD may not be attained anymore. Nevertheless, minimizing sequences may 
admit a limit in some sense. As will be seen at Section O this phenomenon is tightly 
linked to the notion of generalized entropic projection introduced by Csiszar. 

4.1. The results about minimizing sequences. One starts this section stating its 
main results at Theorems 14.11 and 14.71 



Theorem 4.1. The hypotheses of Theorem 2.11 are assumed 



(a) Suppose that C fl domT* 7^ 0. Then, the minimization problem (|PcJ) is attained in 
L' x and all its solutions share the same unique absolutely continuous part Q G L\*R. 



(b) Suppose that C fl icordomT* 7^ 0. Then, ( Dp ) attained in y and 

Q*{dz)=i z {{u^e{z)))R{dz) 



(4.2) 



where oj G y is any solution to (Dg). More, for each such u Q , (a; 0) #(•) ) is in the 
strong closure of T*domT in K\ and there exists u E y solution of ( Dp ) such that 

pr«]? = Ke(.)>. 



Proof. • Proof of (a). The attainment statement is The orem 12. 171 b. Let us show that as 
7* is strictly convex, if k* and are two solutions of (Pc), their absolutely continuous 
parts match: 

k = e (4.3) 
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are in the convex set {£ G L' x ;T£ G C} and inf(Pc) = I(k*) = I(£#). For all 
< p, q < 1 such that p + q = 1, as / and I s are convex functions, we have 

inf(P c ) < I(ph + qQ 

= KpK + qty + rfrK + qti) 

< pI{K) + qI(£:)+ P I s (K) + ql s (£l) 
= pI{K) + qI(Q = inf (P c ) 

It follows that I(pK+q£$)+I s (pk*+q£l) = pl{k*)+ql(£l)+pl s (kl)+ql s (£l). Suppose that 
kl 7^ 1%. As I is strictly convex, with < p < 1, one gets: /(p/c* + gl") < pI{K) + ql{£%) 
and this implies that I s (pk^ + g£j) > pl s (kl) + ql s (£l) which is impossible since I s is 
convex. This proves (14.31) . 

• Proof of (b). Let £ be any solution to (Pc)- Denoting x a = T£ a and x s = T£ s one sees 
with (12131) that 

(a) x a G [C - X s ] n domT* 
(6) (u>,x a ) < (u,x),Vx G [C - x s ] n domT* 
(c) ? = 7i«c5 i e»i2 

By Theorem 12. 171 -c. this implies that u solves (Dc-s s )- 

It remains to show that u also solves (Do)- Thanks to Theorem 12. 171 we have: inf xe c'(a;, x) - 
(T*u>, £) = ((u,9),£ a )_+ I(£ s ) and inf xeC (u;,x) - I y ((Q,9)) = sup(D c ) = inf(P c ) = 
I(£ a ) + I(£ s ) = inf_(P c _ ss ) + I(£ s ) = sup(Dc-s») + I(£ 8 ). Therefore in£ xeC (w,x) = 
xai xe c-xs{u, x) + I(£ s ) and subtracting I(£ s ) from (D c ), one sees that Co which solves 
(Dc-x 3 ) also solves (Dc)- One completes the proof of the proposition, taking u = Co. □ 

Remark 4.4. Replacing £ s with t£ s , the same proof shows that Co solves (Pc+(t-i)x») f° r 
any t > 0. 

From now on, one denotes Q<> G Laj-R the absolutely continuous part shared by all the 
solutions of (Pc). Let us introduce 

C = \qeL k R- TQ:= J 0dQecJ (4.5) 

C = {£eL' Xo ; T£-.= (9,£)eC} 

the constraint sets T~ 1 C n L\*R and T~ 1 C on which / and I are minimized. We have: 
C = C D L\*R and I = I + l l R . Hence, inf(P c ) < inf(P c ). Clearly, C H dom/ ^ <£> 

inf(Pc) < oo implies C n dom I ^ ^> inf(Pc) <oo^>Cn domT* ^ 0. 

Of course, if C Pi dom / 7^ 0, QPcP admits nontrivial minimizing sequences. Theorem 14.71 

below gives some details about them. 

The present paper is concerned with sets C of the form (14.51) . but this is not a restriction 
as explained in the following remarks. 

Remarks 4.6. 

(1) Taking T* to be the identity on y o = E\ or L\ (being careless with a.e. equ ality , 
this corresponds to 9{z) to be the Dirac measure 8 Z ), one sees that (Hi) or pi 
is satisfied respectively. Hence, with C = C, JPcD is ( TPcD - 

Consequently, the specific form with 9 and C adds details to the description of C 
without loss of generality. 

(2) With 9, y o and C as in (1), the assumption (Ac) is: 
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(a) Under §A$: C is a (L K R, £ A J -closed. 

Note that if A and A* both satisfy the A 2 -condition, as C is convex, this is 
equival ent to C is || • ||Aj-closed. 

(b) Under (Ag): C is a(L\*R, L Ao )-closed. 

Note that if A* satisfies the A 2 -condition, as C is convex, this is equivalent to 
C is II • |U*-closed. 

II II 

We denote || ■ || A *-int (C) the interior of C in L\*R with respect to the strong topology 
of Lax- 



Theorem 4.7 (Minimizing sequences of QPcP ). Suppose that the assumptions of Th 



eorem 



2.17 are satisfied and C fl dom / ^ 



Let us consider the following additional conditions. 

(1) a- There are finitely many moment constraints, i.e. X Q = IR X (see Section \3. 2. 1\) 
b- C fl icordom/ ^ 0. 

(2) C is a a(L\*R, L\ )- closed convex set with a nonempty || ■ \\\* -interior. 

Under one of these additional conditions (1) or (2), any minimizing sequence of < \Pg\ > 
converges to Q c with respect to a(L\*R, E^o) an d HQo) — m f(Pc) = i n f(Pc*)- 

Proof. This proof relies on results which are stated and proved in the remainder of the 
present section. It is shown at Lemma H~9l that any minimizing sequence of flP^D converges 
in the sense of the a(L\iR, -EA )-topology to Q , whenever inf(Pc-) = inf(P<7). But this 
equality holds thanks to Lemma [4.191 and 

• under condition (1): Lemma [4.211 a; 

• under condition (2): Corollary 14.231 -b. 

Let us have a look at the last inequality. For any I = £ a + I s = Q + I s minimizer of ( Pg ) 
and any {Q n ) n >i minimizing sequence of flP^D , one obtains 

MI(Q n ) = inf (P c ) = inf (P c ) = 1(1) 

n 

= i(Q ) + r(I s ) 
> HQ.) 

with a strict inequality if I s (I s ) > 0. □ 
Remarks 4.8. 

(1) As regards condition (1), it is not assumed that C has a nonempty interior. 

(2) As regards condition (2): 

(i) Any a(L\*R, Lx a )-c\osed convex set has the form 



C= f) {£EL K R; (u,l)>a y } 



for some U C L A<> ; 



(ii) For C to have a nonempty || ■ ||Aj-interior, it is enough that C fl has a 
nonempty interior in endowed with the uniform dual norm | • |£- This is 
a consequence of [HI Lemma 4.13-e]. 
(3) The last quantity I s (£ s ) = inf (Pc) - I(Qo) is precisely the gap of lower a(L\*R, E\ a )- 
semicontinuity of / : \im n Q n = Q<> and inf(Pc) = liminf n I(Q n ) > I(\\m n Q n ) = 

I(Qo). 



16 



CHRISTIAN LEONARD 



4.2. A preliminary lemma. Preliminary results for the proof of Theorem 14.71 are stated 



below at Lemma [4. 91 The assumption (A# ) about the bad constraint is T*y o C L 



Lemma 4.9. Suppose that the assumptions of Theorem 2.11 are satisfied, inf(Pc) < oo 
and 

inf(P c ) = inf(P c ) (4.10) 
Then, any minimizing sequence of (|P C D converges to Q with respect to a(L x *R, E\ ). 



Proof. Let (Q n ) n >i be a minimizing sequence of flPcD- S ince it is assumed that inf(Pc 



inf(Pc), (Q n )n>i is also a minimizing sequence of (|Pcj). With the representation of I at 
(14.111) below, one can apply ([LZj, Corollary 2.2) which states that / is cr(L' x<! , Lx^-inf- 
compact. Hence, one can extract a o~(L' Xt> , L Ao )-convergent subnet (Q a ) a from {Q n ) n >\. 
Let £.* G C denote its limit: we have lim a f z u dQ a = u) for all u G Lx - As {£%, u) = 0, 
for all u G -E^o (see Theorem 12.21 a). we obtain: lim a J z udQ a = f z ud£1 for all u G E\ . 
This proves that (Q a ) a o~(E' x<> , E Xo )-converges to £®. As -Ea is a separable Banach space 
(Lx is not separable in general), the topology cr(E' x , Ex ) = o~(L\*R, -Ea<>) is metrizable 
and one can extract a convergent subsequence (Qk)k>i from the convergent net (Q a ) a . 
Hence, (Qk)k>i is a subsequence of (Q n )n>i which a{Lx%R, -Eao) -converges to £%. 



Since J is inf-compact, £* is a minimizer of ( Pg ) and by Theorem 14.11 a. there is a unique 



Qo such for any minimizing sequence (Q n ) n >i, £® = Q . Therefore, any convergent subse- 
quence of (Q n ) n >i converges to Q Q for a(L x *R, E x<> ). As any subsequence of a minimizing 
sequence is still a minimizing sequence, we have proved that from any subsequence of 
(Q n )n>i, one can extract a sub-subsequence which converges to Q Q . This proves that 
(Qn)n>i converges to Q with respect to a(L\*R, E^). □ 

4.3. Sufficient conditions for inf(Pc) = inf(P c ). Our aim now is to obtain sufficient 



cond ition s for the identity inf(Pc) = inf(Pc) to hold. Let us rewrite the problems flPgD 



and (|PcJ) hi order to emphasize their differences and analogies. Denote 



= h(u) = J X(u)dR, ueL Xo 

<5> E (u) = $l(m) + l e (u), u G L Xo 
Their convex conjugates are 

& E (t) = sup {(£,u)- h(u)}, £eL x *R 

udE x<> 

<t>l(£) = sup {(£,u) - l x (u)}, teL' x<> 

It is shown in [T7] that 

f I(£) = n(?-mR), £EE' Xo = L K R Uu) 
{!(£) = <f>* L (£-mR), £eL' Xo = L K R®Ll 

Hence, considering the minimization problems 

minimize §* E {£) subject to (9,£) G C Q , £ G L X% R (P E ) 

and 

minimize $* L (£) subject to (9, £) e C , £ G L' x<> (P L ) 
with C a = C — (9,mR), we see that £* is a solution of QPcP [resp. (P_c)] if and only if 



mR is a solution of (Pe) [resp. (Pl)]- It is enough to prove 

inf (P E ) = inf (P^ (4.12) 
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to get inf(P c ) = inf(P c ). 

Basic facts about convex duality. The proof of (14.121) will rely on standard convex 
duality considerations. Let us recall some related facts, as developed in [26J. 
Let A and X be two vector spaces, h : A — > [—00, +00] a convex function, T:i->Ia 
linear operator and C a convex subset of X. The primal problem to be considered is the 
following convex minimization problem 

minimize h(a) subject to Ta G C, a G A (V) 

The primal value-function corresponding to the Fenchel perturbation F(a,x) = h(a) + 
Lc(Ta + x), a G A, x G X is (p(x) = inf agA F(a, x), i.e. 

<p(x) = inf{/i(a); a G A, Ta G C — x} x G X. 

Denote A* the algebraic dual space of A. The convex conjugate of h with respect to the 
dual pairing (A, A*) is 

h*(u) = sup{(u, a) — h(a)}, v G A*. 

Let Y be a vector space in dual pairing with X. The adjoint operator T* : Y — > A* of T 
is defined for all y G Y by 

(T*y, a) A *,A = (y, Ta) Y ,x, Va G A 
The dual problem associated with (V) is 

maximize inf (?/, 2) — h*(T*y), y EY {T>) 

Let U be some subspace of A*. The dual value-function is 

ip(u) = sup < inf (y, x) — h*(T*y + u) > , u G U 

y &Y t^GC J 

One says that (X, Y) is a topological dual pairing if X and F are topological vector 
spaces and their topological dual spaces X' and Y' satisfy X' = Y and Y' = X up to 
some isomorphisms. 

Theorem 4.13 (Criteria for the dual equality). We assume that A, U, X and Y are locally 
convex Hausdorff topological vector spaces such that (A, U) and (X, Y) are topological dual 
pairings. For the dual equality 

inf (V) = sup(£>) 

to hold, it is enough that 

(1) (a) h is a convex function and C is a convex subset of X, 

(b) (p is lower semicontinuous at G X and 

(c) sup(X?) > —00 

or 

(2) (a) h is a convex function and C is a closed convex subset of X, 

(b) ip is upper semicontinuous at G U and 

(c) inf (V) < +00. 

Remarks 4.14. About the space U. 

(a) As regards Criterion (1), the space U is unnecessary. 

(b) As regards Criterion (2), it is not assumed that T*Y C U. 
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Back to our problem. Let us particularize this framewor k fo r the problems (Pe) and 
(P L )._ Assuming that m = 0, one sees that $P~c\)=(Pe), §Pc§ = (Pl), I = I\* = $* E , 
I = I\* = &* L , ^ = X, C = C Q and so on. This simplifying requirement will be assumed 
during the proof without loss of generality, see the proof of [151 Theorem 4.6]. 

Let us first apply the criterion (1) of Theorem 14.131 
Problem (Pe) is obtained with A = L X *R, X = X L , Y = y L equipped with the weak 
topologies o~(X L ,y L ) and a(y L ,X L ), and h = §* E = I. The corresponding primal value- 
function is 

<p E (x) = M ^I(Q); J 9dQ E C - x,Q E L x *rX , x E X L 

Under the underlying assumption (Ag), with (HJ Lemma 4.13-g] we have: 

T*y L C L\ . (4.15) 

Hence, one only needs to compute h* on L x<> C [L X *R\* = A*. For each u E L x<> , h*(u) = 
supj-g^, {f z uf dR — f z X*(f) dR} and it is proved in [25] that 



h*(u) = j X(u)dR, uEL Xo (4.16) 
Therefore, the dual problem associated to (Pe) is 



maximize inf(y,a;)— / X(T*y)dR, y E yL (D-e) 
x ^ c Jz 

Let us go on with (P_l). Take A = L' Xr> , X = X^, Y = y^ equipped with the weak 
topologies cr(X L ,y L ) and cr(yL,X L ), and h = §* L = I. The function h* in restriction to 
L Xt> is still given by (14.161) since u E L Xt> i— > f z X(u) dR is closed convex (Fatou's lemma). 
The primal value-function is 

<p L (x) = inf {!(£)■ (6,£}eC-x,£e L' x J, x e X l 

and the dual problem associated to (Pl) is 

(Dl) = (Pe). 

Lemma 4.17. Suppose that T*y L C L Xo and C fl X L is o~(X L ,y L )-closed, then ip L is 
v^Xl^l) -lower semicontinuous. 

Proof. Defining <p(x) := (fi(—x) and J(x) := inf{/(£);£ E L' Xo : (9,£) = x}, x E X L , we 
obtain that (p is the inf- convolution of J and the convex indicator of —C : i_c- That is 
(p(x) = (JUl_ c )(x) =ini{J(y) + L^ c (z);y,z,y + z = x}. 

As already seen, I is cr(L' x<> , LA )-inf-compact and T is o~(L x<> , L x<! )-o-(Xl, 3^)-continuous, 
see [IH Lemma 4.13-h]. It follows that J is a(X L , 0^)-inf-compact. As Cfl X L is assumed 
to be ct(Xl, 3^L)-closed, i-c is lower semicontinuous. Finally, being the inf-convolution of 
an inf-compact function and a lower semicontinuous function, (p is lower semicontinuous, 
and so is ipL- □ 

As / and C are assumed to be convex and sup(D^) > inf xg c(0,x) — j z A(T*0) dR = 
> — oo, this lemma allows us to apply Criterion (1) of Theorem 14. 131 to obtain 

inf(P L ) = sup(D L ) (4.18) 

Since I and I match on L X *R, we have inf (P^) < inf(P^). Putting together these consid- 
erations gives us 

sup(D i; ) = sup(D L ) = inf(P L ) < M(P E ). 
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Since the desired equality (14.101) is equivalent to inf(P L ) = inf(P£), we have proved 
Lemma 4.19. The equality holds if and only if we have the dual equality 

inf(P B ) = sup(D B ). (4.20) 
This happens if and only if fE is c(Xl, y^-lower semicontinuous at x = 0. 

Let us now give a couple of simple criteria for this property to be realized. 
Lemma 4.21. 

(a) Suppose that there are finitely many constraints (i.e. X Q is finite dimensional) and 
C R icordom/ ^ , then ipE is continuous at 0. 



(b) Suppose that (A%) is satisfied and C is o-(X E ,yE)-closed, then ip E is a(X L ,y L )-lower 



semicontinuous. 



Proof. • Proof of (a). To get (a), remark that a convex function on a finite dimensional 
space is lower semicontinuous on the intrinsic core of its effective domain. By [T4"l Lemma 
4.13], T is a(L\*R, LA )-continuous and the assumption C fl icordom/ ^ implies that 
belongs to icordom^^. 



Proof of (b). It is similar to the proof of Lemma 14.171 The assumption (A^) insures 



that X E = X L and T is cr(L A j R, E\ )-a(X E , 34;)-continuous. □ 

It follows from Lemma [4.211 b. Lemma 14. 19[ the remark at (14.121) and Lemma [4.91 that 
under the good constraint assumptions (^4 V ), if C n domT* ^ 0, then any minimizing 
sequence of QPcD converges with respect to the topology a(L\*R, Ex^) to the unique 



solution Q of QPc[ ). This is Theorem 12. 141 b. 



Using Criterion (2). Up to now, we only used Criterion (1) of Theorem 14.131 In 
the following lines, we are going to use Criterion (2) to prove (I4.20p under additional 
assumptions. 

Let us go back to Problem (P_b). It is still assumed without loss of generality that 
m = and 7 = A. One introduces a space U and a dual value-function ip on U. The 
framework of Theorem 14.131 is preserved when taking X = X^, Y = with the weak 
topologies c(X L ,y L ) and a(y L ,X L ), h = I on A = L\*R as before and adding the 
following topological pairing (A, U). We endow A = L\*R with the topology a(L x *R, L Xt> ) 
and take 

U = A' = (Lx*R)' * L\ 
with the topology a{L\ 0) L\*). By (14. 16[) : h* = I\, this leads to the dual value-function 

ip(u) = sup <^ inf (y, x) - I\(T*y + u) \ , u G L Xo 



yey L 



To apply Criterion (2), let us establish the following 

Lemma 4.22. For ip to be a{L\ a1 L\*)-upper semicontinuous, it is enough that 

(a) T*y L is a cr(L A<> , L\*)-closed subspace of L Xo and 

(b) the interior of C in L\*R with respect to || ■ \\\* is nonempty. 
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Proof. During this proof, unless specified the topology on L Xt> is o"(L Ao , L\*). For all u G 

-i>(u) = inf \suv{(-T*y,e);e eL K R: Tie C} + I x (T*y + u)} 
= inf |sup{(-t;,£);£GC} + / A (t; + M)) 

where V = T*y L and I\OG(u) = mi veLx ^{G(v) + I\{u — v)} is the inf-convolution of I\ 
and G{u) = l* c {u) + Ly(u),u G La<>- Let us show that under the assumption (a), 

G = l* c 

As V is assumed to be closed, we have iy = iy ± with V 1 - = {k G L\*R; (v, k) = 0, Vt> G 
V}. This gives for each it G Z/ A », G(tt) = + L v ± ( u ) = su Pe£c( u ^) + su Vkev ± ( u ^ ^) = 
sup{(w, £ + k); £ £ C, k £ V^} = l* c , v x{u) = l* c {u), where the last equality holds since 
C + V ± = C, note that kerT = V ± . 

As convex conjugates, i* c and 1\ = h* are closed convex functions. 
Since for all u, v G L Ao , 

+/a(«- u) = t-c-k( v ) ~ ( v ' k ) + h(u-v) 

= i* c ~k( v ) + (u-v,k) + I x (u - v) - (u, k), 

we have 

-V + <-,fc> = *S_ fc n(/A -(-,*» 

But, by the assumption (b) there exists some k in L^i? such that G int (C — k). It 
follows that i£_ fc is inf-compact. Finally, — ip + (-,k) is lower semicontinuous, being the 
inf-convolution of a lower semicontinuous and an inf-compact functions. □ 

Corollary 4.23. 

(a) Assume that C is a(X L ,y L )-closed convex, T*y L is a o~(L\ , L\*)-closed subspace of 
L Xt> , C has a nonempty \\ ■ \\\*-interior and inf(P^) < oo. Then, U^20) is satisfied. 



(b) In particular, if C is o~(L\*R, L\ )-closed convex set with a nonempty || ■ \\\*-interior 
and inf(P E ) < oo, then §JJZW is satisfied. 



Proof. • Proof of (a). Apply the criterion (2) of Theorem 14.131 with Lemma [4.221 

• Proof of (b). This is (a) with = Lx^ and T* = Id, taking advantage of Remarks 

mi □ 

5. Entropic projections 
The results of the preceding sections are translated in terms of entropic projections. 
5.1. Generalized entropic projections. We consider the convex problem 

minimize I(Q) subject to Q G C (Vc) 

where C is a convex subset of Mz- 

Let (/ n )n>i be a sequence of measurable functions. Following [4], one says that it 
converges i?-loosely in measure to / if for each measurable set A G Z such that R(A) < oo 
and all e > 0, lim„^oo R(A H {\f n - f \ > e} = 0. 

It converges a(L\*R, E^JToosely if for each measurable set A G Z such that R(A) < oo, 
Ia/w converges to 1^/ with respect to a(L\*R, E Xt> ). 
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It converges Z^-loosely if for each measurable set A e Z such that R(A) < oo, IaJu 
converges to l^f strongly in Li(R). 

Definitions 5.1. Let Q and (Q n ) n >i in Mz be absolutely continuous with respect to R. 

(1) One says that (Q n )n>i converges R-loosely in measure to Q if converges R- 
loosely in measure to 

(2) One says that (Q n ) n >i converges a(L x *R, E Xo ) -loosely to Q in L X *R if con- 
verges a(L K R,E x J-loosely to f|. 

(3) One says that (Q n )n>i converges R-loosely in variation to Q if converges L\- 
loosely to 

Definition 5.2 (Generalized entropic projection). p[J Csiszar]. Suppose thatCddoml ^ 
and that any minimizing sequence of the problem (\Vc\j converges R-loosely in measure to 
some G M z . 

This Q* is called the generalized entropic projection of mR on C with respect to I. It may 
not belong to C. In case Q* is in C, it is called the entropic projection of mR on C. 

Theorem 5.3 (Csiszar, |3j). Suppose that (A R ) and (A 7 ») are satisfied. Then, mR has 
a generalized projection on any convex subset C of Mz such that C n dom I ^ 0. 

In jl] 7* doesn't depend on the variable z, but the proof remains unchanged with a 
^-dependence. 

Proposition 5.4. Let Q and (Q n ) n >i in Mz be absolutely continuous with respect to R 
such that (Q n ) n >i is a minimizing sequence which converges loosely in R-measure to Q. 
Then, 

(a) {Qn)n>i converges R-loosely in variation to Q; 

(b) (Q n )n>i converges cr(L x *R, E Xo )-loosely to Q. 

Proof. • Proof of (a). Because of de la Vallee Poussin's theorem (using assumption (A!*)), 
any minimizing sequence (Q n )n>i is such that {lA^^,n > 1} is uniformly integrable for 
each A such that R(A) < oo. Therefore, if (Q n )n>i converges loosely in .R-measure, it 
converges -R-loosely in variation. 

• Proof of (b). It is enough to consider a bounded measure R. If A* is a finite function, 
this follows from [2U Proposition 4.5.6]. If dom A* is bounded, we have to show the 
0"(Loo, Li)-convergence of /„ = to / = But (f n ) n >i is bounded in L^ and the 
result follows. □ 

As a direct consequence of Theorem 12.141 if the constraints are good, the generalized 
entropic projection is the entropic projection. 

Proposition 5.5. Suppose that (An) and (A 7 *) hold, C is convex and a (L X *R, E Xt> )- closed 
and C R dom J ^ 0. Then, the entropic projection Q* exists and is equal to 

Q* = QeC 

where Q is the minimizer of (|PcD which is described at Theorem \2.14 



Proof. This is an easy corollary of Theorem 12.141 □ 

A consequence of Proposition 15.51 is that for any a(L x *R, £' Ao )-closed convex set C, the 
generalized entropic projection is the entropic projection. This is essentially [4, Thm 
3-(iii)]. 
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Csiszar's proof of Theorem 15.31 is based on a parallelogram identity which allows to 
show that any minimizing sequence is a Cauchy sequence. This result is general but it 
doesn't tell much about the nature of Q*. Let us give some details on the generalized 
entropic projections in specific situations. 



Theorem 5.6. Suppose that the assumptions of Theorem 2.17 are satisfied and C given 
by H4.5I) satisfies C R dom J ^ 0. Let us consider the additional conditions: 

(1) a- There are finitely many moment constraints, i.e. X = M> K (see Section \3. 2. 1\) 
b- C PI icordom/ ^ 0; 

(2) C is a a(L\*R, L\ )-closed convex set with a nonempty \\ ■ \\\* -interior. 

Then, under one of the conditions (1) or (2), the generalized entropic projection of 
mR on C is 

the absolutely continuous component described at \J^.2^ and I(Q*) < infci. 

Proof. This is a direct consequence of Theorems 15.31 and 14.71 □ 

5.2. The special case of relative entropy. The relative entropy I(P\R) and its ex- 
tension I(£\R) are described at Section [3j The minimization problem is 



minimize I (P\R) subject to J9dPeC, P G P z (5.7) 
and its extension is 

minimize I(£\R) subject to (9, £) G C, £ G £(Z) (5.8) 
We introduce the Cramer transform of the image law of R by 9 on X Q : 

E(x) = sup | (y,x) - log / e {y ' e) dR \ G [0, oo], x G X (5.9) 
yey D I Jz ) 

Proposition 5.10 (Relative entropy subject to good constraints). Let us assume that 9 
satisfies the u good constraint" assumption 

Vy G y o , J e^ B{z)) R{dz) < oo (5.11) 

and that C D X E is a cr(X E , y E )-closed convex subset of X E . 

(a) The following dual equality holds: 

mi{I{P\R)- (9, P) G C, P G P z } = sup j inf (y, x) - log / e M dR} G [0, oo] 

yey a l x ^ c Jz J 

(b) Suppose that in addition C PI domS ^ 0. Then, the minimization problem ( [5. 7| ) has a 
unique solution P in Pz, P is the entropic projection of R onC = {P G Pz, J z 9dP G 

c}. 

(c) Suppose that in addition, C flicordomS ^ 0, then there exists some linear form uj on 
X Q such that (u, 9) is measurable and 

x 4 J z 9dP G CndomH 
(uj, x) < (a), x),Wx G C(1 dom H 
P(dz) = exp ((a), 9{z)) - log j z dR) R(dz). 
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In this situation, x minimizes H on C, I(P \ R) = E(x) and 

x = J 9(z)exp (@,0(z)) -log J e^ 0) dR^j R(dz) (5.12) 
in the weak sense. 

Proposition 5.13 (Relative entropy subject to bad constraints). Let us assume that 9 
satisfies the "bad constraint" assumption 

My G y o , 3a > 0, J e a|M(2))l R(dz) < oo (5.14) 

and that C fl Xl is a <j(Xl, y 'i)- closed convex subset of X^. 

(a) The following dual equality holds: 

iai{I(£ | R); (9, £) e C,i E £(Z)} = sup j inf (y, x) - log / e {y ' e) dR} G [0, oo] 

yey [ xeC Jz ) 

(b) Suppose that in addition C fl domH ^ 0. Then, the minimization problem $5.8\) is 
attained in S(Z) : the set of minimizers is nonempty, convex and a (L' T , L T )- compact. 
Moreover, all the minimizers share the same unique absolutely continuous part P<> G 
Pz H L T *R which is the generalized entropic projection of R on C. 

(c) Suppose that in addition, one of the following conditions 

(1) X = R K and C n icordom / ^ or 

(2) || • || r .-int(C) ^0. 

is satisfied. Then, there exists a linear form uj on X such that (u, 9) is measurable, 
f z e^' e) dR < oo and 

P (dz) = exp ((&, 9{z)) - log J e { "' 0) dR^j R(dz). 

In Proposition 15.101 x is the dominating point in the sense of Ney (see Definition 16.31) 
of C with respect to S. The representation of x has already been obtained for C with 
a nonempty topological interior in W 1 by Ney in [23j and in a Banach space setting by 
Einmahl and Kuelbs in [8]. The representation of the generalized projection P is obtained 
with a very different proof by Csiszar [2] and ([3], Thm 3). Proposition 15.131 also extends 
corresponding results of Kuelbs [13] which are obtained in a Banach space setting. 
For more details about the minimizers of (15.81) . one can look at ([19], Theorem 3.4) where 
a characterization is obtained under the weakest assumption: C fl domH ^ 0. 

Proof of Propositions \5.10\ and \5.13[ They are direct consequences of Theorems 12 . 141 [2 . 1 71 
Proposition 15.51 Theorem 15.61 and Lemma \5. 151 below. Note that A which is given at (13.11) 
satisfies the condition (ii) of Proposition 12.201 □ 

The following lemma allows to apply the results of the present paper with j(s) = e s — 1 
and the extended constraint ((1,9),£) G {1} x C : the first component of the constraint 
insures the unit mass (1,£) = 1, to obtain results in terms of log-Laplace transform. 

Lemma 5.15. For all x G X Q , 

sup \(y,x) -log / e {y ' e) dR\= sup \(y,(l,x))- [ (e^ (1 ' 0)) - 1) dR \ G (-oo, +oo]. 
yey I Jz J ymxy a [ Jz J 
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Proof. Using the identity: — log b = sup a {a + 1 — be a }, one gets: 



sup 



(y, x) - log / e {y ' e) dR 



sup (y, x) + a + 1 



e M dR 



sup < ({a,y),{l,x)) - / e M+a dR + 1 
am,y&y I Jz 



sup < (y,(l,x)) - / {e {ui ' e)) - 1) dR 
ymxy a { Jz 



which is the desired result. 



□ 



Example 5.16. Csiszdr's example. Comparing Proposition [531 with Theorem l5.6l one may 
wonder if the a(L\*R, .E^J-closedness of the convex set C is critical for the existence of 
an entropic projection. The answer is affirmative. In [3j Example 3.2], Csiszar gives an 
interesting example where the generalized entropic projection can be explicitly computed 
in a situation where < \Pq\) is n °t attained, see also [H Exercise 7.3.11]. This example is 
the following one. 

Take the probability measure on Z = [0, oo) defined by R(dz) = aojr^sdz where ao is 
the normalizing constant, / the relative entropy with respect to R, 9 : z G [0, oo) i— > z G 
X = R and C = [c,oo). This gives C = {Q G L T *R; L ) zQ{dz) > c,Q([0,oo)) = 1}. 
The point is that 6 is in L T (R) but not in E T (R). 

By Theorem I5.13[ the generalized projection of R on C exists and is equal to P v = 



"y i- 



-g- dz for some real y < 1, where a y is the normalizing constant. If it belongs to C, 
then J z zP y (dz) > c. But sup y<1 f 2 z P y {dz) = f z zPi(dz) = a\ § z jT^dz := x* < oo. 
Therefore, for any c > x*, there are no entropic projection but only a generalized one. 
A detailed analysis of this example in terms of singular component is done by Leonard and 
Najim [SUJ Proposition 3.9]. This example corresponds to a a(L\*R, LA )-closed convex 
set C such that no entropic projection exists but only a generalized one. 
More details about this example are given below at Example 16. 101 



6. Dominating points 

The underlying assumptions are Theorem I2.17f s ones. We are going to investigate 
some relations between dominating points and entropic projections. In the case where the 
constraint is good, Theorem 12.141 and Proposition 15.51 state that the generalized entropic 
projection is the entropic projection = Q, the minimizer x is the dominating point of 
C, it is related to by the identity: 

x=(e,Q*). (6.1) 

We now look at the situation where the constraint is bad. As remarked at Example 15.161 
the above equality may fail. A necessary and sufficient condition (in terms of the function 
r*) for x to satisfy (16.11) is obtained at Theorem 16. 81 

Following Ney [22j [23] , let us introduce the following definition. A point x G X sharing 
the properties (a), (b) and (12.161) of Theorem 12.141 is called a dominating point. 

Definition 6.2 (Dominating point). Let C C X Q be a convex set such that C D Xl is 
a(XL,yi)-closed. The point x G X Q is called a r*-dominating point if 

(a) x G CndomT* 

(b) there exists some linear form Co on X such that (oj,x) < (oj,x) for all x G C H 
domT*, 
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(c) (uj, 9 (•)) is measurable and x = J z 9(z)j'(z, (uj,9(z))) R(dz). 

In the special case where T* is replaced by the Cramer transform 5 defined at (15.91) . 
taking Lemma [5 . 1 51 into account, this definition becomes the following one. 

Definition 6.3. Let C C X be a convex set such that C fl X L is a (X L ,y L ) -closed. The 
point x G X Q is called a S-dominating point if 

(a) x G C fl dom H 

(b) there exists some linear form uj on X Q such that (u),x) < (u,x), for all x G 
C fl dom H and 

exp((u,9(z)}) 



c) (u), #(■)) is measurable and x = j 9{z) R(dz) where Z(u>) is the unit 



mass normalizing constant. 



z 



Note that this definition is slightly different from the ones proposed by Ney [23] and 
Einmahl and Kuelbs [8] since C is neither supposed to be an open set nor to have a 
non-empty interior and x is not assumed to be a boundary point of C. The above integral 
representation (c) is f)5.12p . 

Recall that the extended entropy I is given by lETTj) : I{£) = I(l a )+I s (£ s ), i = t +£ s G 
L' x<> and define for all x G X Q , 

J(x) ± M{I(£);ieL' Xo ,(9,£)=x} 
J(x) ^ M{I(£);£eL K R,(9,t) = x} 
J s (x) ± irf{I s (£);£eLl,(9J)=x} 
Because of the decomposition L' x<> ~ L\*R © L s x<> , one obtains for all x G X Q , 
J(x) = M{I(£ 1 ) + r(£ 2 );£ 1 eL x *R,£ 2 eLl,(9,£ 1 + £ 2 )=x} 
= inf{ J(xi) + J s (x 2 ); xi, x 2 G X a , x± + x 2 = x} 
= Jaj s (x) 

where JOj s is the inf-convolution of J and J s . By Theorem 12.171 -a. J = T* and if 
J(x) < oo, there exists £ x G L' Xo such that (9,£ x ) = x and J(x) = I{£ x ). Let us define 

x a :=(9,t x ) and x s :=(0,^). 

These definitions make sense since £% is the unique (common) absolutely continuous part 
of the minimizers of I on the closed convex set {£ G L' x<> ; (9,£) = x}, see Theorem 14.11 -a. 
Of course, we have 

x = x a + x s 

and as J{x) = I(£ x ) = I(£ a x ) + P(t x ) > J(x a ) + J s (x s ) > Ja.J s (x) = J{x), one gets the 
following result. 

Proposition 6.4. For all x G dom J, we have: 

J(x) = J(x a ) + J s (x s ), J(x a ) = I{£ a x ) and J s (x s ) = I s (£ s x ). 

Now, let us have a look at the dual equalities. The recession function of T* is defined 
for all x by 

f*(x) = lim T*(tx)/t G (-oo,+oo]. 

t— >+oo 

Definition 6.5 (Recessive x). Let us say that x is recessive for T* if for some 5 > and 
£ G X OJ T*(x + t£) — T*(x) = £T*(£) for all t G (—5, +oo). is said to be non-recessive 
otherwise. 
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Proposition 6.6. We have J = T* and J s = T*. Moreover, J(x) = T*(x) for all non- 
recessive x £ X . 

Proof. We have already noted that J = T* and by [HI Theorem 2.6-a], we get: J s = 
'-domr : the support function of domT. Therefore, it is also the recession function of T*. 
Hence, we have T* = Jj= J^J 8 = JUT*. 

Comparing T* = JOT* with the general identity T* = T*OT*, one obtains that J(x) = 
T*(x), for all non- recessive x £ X D . □ 

Proposition 6.7. For all x £ domT*, we have: 

T*(x) = T*(x a ) +f ; (x s ). 

Moreover, x is non-recessive if and only if x s = 0. In particular, x a is non-recessive. 

Proof. By (12. 7p . we have I(£ x ) = + I(&x) w here / is the recession function of I. It 
follows that J(x) = J(x a ) + J s (x s ), since J(x a ) = (Proposition 16. 41) and the recession 
function of J is J s . To show this, note that (see [27j) 

- I s is the recession function of I, 

- the epigraph of x i— > mi{f(£); £, Ti = x} (with T a linear operator) is "essentially" 
a linear projection of the epigraph of /, (let us call it an inf-projection) 

- the epigraph of the recession function is the recession cone of the epigraph and 

- the inf-projection of a recession cone is the recession cone of the inf-projection. 

The first result now follows from J = T*. The same set of arguments also yields the second 
statement. □ 



Theorem 6.8. Let us assume that the hypotheses of Theorem \2.11\ hold and that C D 
icordomT* ^ 0. 

(a) Then, a minimizer x ofT* on the set C is a V* -dominating point of C if and only 
if x is non-recessive. This is also equi vale nt to the following statement: "all the 



solutions of the minimization problem (jPp ) are absolutely continuous with respect 



to R. " In such a case the solution of (|PcJ) is unique and it matches the solution 
of O- _ 

(b) In particular when V* admits a degenerate recession function, i.e. T*(x) = +oo 
for all x 7^ 0, then the minimizer x is a T* -dominating point of C. 

(c) The same statements hold with V* replaced by 5. 

Proof. This is a direct consequence of Theorem 12. 17[ Proposition 16.71 and Lemma f5. 151 □ 

Remark 6.9. A remark about the steepness of the log-Laplace transform. In |T31 Thm 1], 
with the setting of Section I5T21 where X Q is a Banach space, Kuelbs proves a result that is 
slightly different from statement (b) of the above theorem. It is proved that the existence 
of a S-dominating point for all convex sets C with a nonempty topological interior is 
equivalent to some property of the Gateaux derivative of the log-Laplace transform y £ 
X' Q i — ► log f x exp((y,x))Ro 9" 1 (dx) on the boundary of its domain. This property is 
an infinite dimensional analogue of the steepness of the log-Laplace transform. It turns 
out that it is equivalent to the following assumption: the Cramer transform H admits a 
degenerate recession function. 

Example 6.10. Csiszdr's example continued. We go on with Example 15.161 By Lemma 
I5.15[ T* is the Cramer transform of R (identifying (l,x) with x). This means that V* 
is the convex conjugate of the log-Laplace transform Y{y) = log Jj Q ^ a 6 ^^ dz, y £ R. 
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Clearly, domr = (— oo, 1] and ) = Jj Q ^ z Pi(dz) := x* < oo. It follows that for 

all x > x*, — r*(x*) = x — x*. One deduces from this that (x*,oo) is a set of 

recessive points. By Theorem 16.81 they cannot be dominating points. Note also that the 
log-Laplace transform T is not steep. 
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